DOCUMENT RESUME 



ED 455 149 



SO 031 951 



AUTHOR 

TITLE 

INSTITUTION 
PUB DATE 
NOTE 



AVAILABLE FROM 

PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Church, Terrence 

System-Wide Assessment in the Arts: A Developing Model. 
Australian Inst, of Art Education, Melbourne. 

1999-09-00 

8p . ; Paper presented at the Annual World Congress of the 
International Society for Education through Art (InSEA) 

(30th, Brisbane, Australia, September 21-26, 1999) . This 
paper has been assisted by the Commonwealth Government 
through the Australian Council for the Arts, its art funding 
and advisory body. 

Australian Institute of Art Education, C/Suite 125, 283 
Glenhuntley Road, Eisternwick, VIC 3185, Australia. 

Reports - Evaluative (142) -- Speeches/Meeting Papers (150) 
MF01/PC01 Plus Postage. 

Elementary Secondary Education; *Evaluation Methods; 
Evaluation Research; Foreign Countries; Models; *Test 
Construction; *Tests; *Visual Arts 

♦Artistic Evaluation; *Australia (Western Australia) 



ABSTRACT 



This paper discusses the development and implementation of 
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the background to the assessment, the design brief, and the assessment 
package as a whole, focusing on the visual arts tests. It discusses issues 
that arose during the development of the test and outcomes highlighted in the 
data analysis. (BT) 
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SYSTEM-WIDE ASSESSMENT 
IN THE ARTS: A DEVELOPING MODEL 

Terrence Church, Murdoch University, Perth, Western Australia 



INTRODUCTION 

The purpose of this paper is to discuss the development and implementation of the system-wide assessment 
of arts learning in primary and secondary schools which took place in Western Australia, in September, 
1996. A description is given of the background to the assessment, the design brief and assessment package 
as a whole with particular focus on, and examples drawn from the visual arts tests. Mention is made some of 
the interesting issues that arose during the process of test development and also of some of the outcomes 
highlighted in the analysis of data. 

It is believed that at the time of the development and implementation of this assessment no other similar 
system-wide arts assessment had been conducted anywhere in the world on such a scale. There had been 
some small scale testing, but the only promise of anything significant was from the United States where a 
team from the Educational Testing Service in Princeton was doing preliminary research for a National 
Assessment of Educational Progress (NAEP) of some of the arts disciplines. This assessment program has 
since been completed, albeit greatly reduced from its original intentions (National Centre for Education 
Statistics, 1998). 

BACKGROUND 

The Monitoring Standards Branch within the Education Department of Western Australia has regularly 
administered system-wide assessments of learning in Western Australian schools, most commonly in 
Mathematics and English. However, assessments have also been conducted in Science, Society and 
Environment, Physical Education and on a small scale, Technology. Results of these assessments are used 
by the Education Department to inform future directions in education planning. It recognises that all eight 
learning areas must be assessed on a rotation basis, although it has a policy of assessing Mathematics and 
English more regularly than the other areas. A Monitoring Standards Assessment in the Arts in 1996 was 
seen as an adventurous step into the unknown for the Education Department of Western Australia. It was 
one that was welcomed whole heartedly by arts educators from every arts discipline involved. 

The contract to develop the tests was awarded to a consortium of people from Murdoch University in 
Western Australia and the Educational Testing Centre at the University of New South Wales. This 
consortium was responsible for the design and production of all test items, all the teacher support material, 
the marking keys for each test and the analysis of data using the Rasch Extended Logistic Model (Mercer & 
Church, 1998). My role on this team was that of the arts consultant. 

THE DESIGN BRIEF 

The Monitoring Standards Branch of the Education Department wanted assessment of all five disciplines 
included in the Western Australian Curriculum Framework for arts learning. These disciplines are Dance, 
Drama, Media, Music and Visual Art. It was agreed that because this was the first attempt at system-wide 
testing in the arts in Western Australia, and that because all of the arts were to be assessed, it would be 
advisable to attempt to limit the scope of the assessment. The design team, together with Monitoring 
Standards personnel, decided to take a ‘broad brush stroke’ approach and design items which touched upon 
a wide range of arts elements, rather than any particular focus which could be examined in depth. Another 
important part of the brief was that as an outcome of the data analysis, student achievement had to be 
assigned levels from the Student Outcomes Framework for Western Australia. To do this all four Major 
Outcomes for the Arts Learning Area in Western Australia had to be addressed through the assessment. 
These outcomes are similar to those of other Australian states. Two of them address art making and the 
other two address art understanding. They are: 

1 . Arts Ideas: Students generate arts works that communicate ideas. 

2. Arts Skills and Processes: Students use skills, techniques, processes, 
conventions and technologies of the arts. 

3. Arts Responses: Students use their aesthetic understanding to respond to, 
reflect on and evaluate the arts. 

4. Arts in Society: Students understand the role of the arts in society. 
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(Curriculum Council of Western Australia, 1998) 



TEST DESIGN 

For each of the disciplines the tests followed the same design, being guided by the following overarching 
principle: to gain a good indication of student knowledge of an art form it is essential to assess both the 
student ability to practice that art, that is, the ’doing of it’ or ‘making of it’, and the students’ ability to 
‘understand’ and ‘appreciate’ that art form. This meant, therefore, that each student had to complete two 
tests, a practical test to assess students’ ability to make art - a Process test and a pencil and paper test to 
measure understanding and appreciation of art - an Analysis Test 

In order that data from the tests could be used to place student achievement on an outcomes continuum, it 
was essential that the tests be designed in a way to facilitate comparable standards across the three year 
levels. To do this common test items were used to link each of the year levels. In the Analysis Test links 
were made through the stimulus material. 

To provide further links, some of the items from the year three test were repeated in the year seven and ten 
test and some of the items from the year seven test, relating to the reproduction, Tamil/’, were repeated in 
the year ten test. Hence in the marking of these link items it was possible to place the responses on a 
continuum of outcome levels. One link item that appears in ail three tests follows. It refers to the 
reproduction, “Waiting For the Broth”. It is in two parts with part a) a multiple choice and part b) a free 
response item asking students to justify their choice in part a). 

a) Choose one word that best describes the use of space in the picture. 

empty crowded open spaced 

b) What is it in the picture that helped you make your choice? 



In the Process Tests links were made through the marking of art making skills. The same marking key was 
used for all three levels. An example of a marking key for one of items follows: 

To what extent does the student’s work reflect the set stimulus/subject? 

0 marks No development - ignored set topic; no reflection of topic; unrelated 

images 

1 mark Beginning to develop - shows little imaginative connection with topic; 

has used some idea(s) as a starting point, but shows no development; 
uses schematic images 

2 marks Sound development - artwork remains on topic, but shows little 

imagination or depth of understanding about topic; reveals some 
thought; has cliche images 

3 marks Well developed - artwork shows imaginative exploration of topic; has 

interesting detail related to topic; some images venture away from the 
obvious/clich6 

4 marks Highly developed - artwork subject(s) is thought provoking and highly 

imaginative; has detail which heightens meaning; has something 
different to say about topic 

There were two other ways that links were made between the three year levels in the process tests. Apart 
from assessing student final art products, we believed it appropriate to monitor the students’ efforts at two 
other points. We asked students to provide written or diagrammatic planning and a written critique of their 
finished work. Both of these were assessed as separate test items. Student responses to these items proved 
to be a disappointment. Even though we knew that in the teaching and learning of most art forms at the year 
levels being assessed, students were not normally required to plan and evaluate on paper, we wanted to 
give it a try in order to provide further link items between year levels and to give deeper insight into the 
students’ ability to make and reflect upon their own art. It was obvious that the students at all levels and in all 
art forms found it difficult to complete these items. 

We were very aware of the fact that we were designing assessment tools for disciplines with which many 
students would have had little or no experience. Likewise we were asking teachers, especially of years three 
and seven students, to administer tests in an area they probably had not taught. This is particularly the case 
with dance and media. Even in the more commonly experienced areas of drama, music and visual art, we 
knew that the teaching and learning is in some instances may be spasmodic. Consequently we sought to 
design tests that teachers would feel comfortable in administering and that students would not experience as 



a failing experience. With these considerations in mind we designed the items in each of the Analysis Tests 
largely as open ended questions to allow students to respond freely at their own level. For example: 

Q. What materials did the artist use to make this picture? 

Q. What can you say about how the materials are used? 

With the Process Tests, it was equally important to design tests that did not give students a failing 
experience. We decided that these tests had to do several things apart from test performance in the art. 
They had to provide a teaching and learning experience in the process of making art, give instructions for the 
implementation of the tests that would be acceptable for both experienced and inexperienced teachers and, 
finally, ensure that the test implementation be standardized. The teachers were given a scripted set of 
instructions. In the case of the performance arts these instructions took the students through a mini lesson 
exposing the students to some of the elements of the discipline. This lesson, and associated instructions, 
gave students a clear idea of what was expected of them. In the case of the non-performance arts, media 
and visual art, teachers gave students scripted instructions together with a list of criteria which students 
might find helpful. By providing a set script for the teacher, this served to assist those teachers whose arts 
experience may have been limited and it also assisted in standardizing the assessment. 

PROCESS OF TEST DEVELOPMENT 

The process of test development was conducted in a most rigorous way from the beginning of the project. 
There was some consultation with The Educational Testing Service in Princeton which was planning an 
assessment to use a model of analysis and portfolio test forms. Many meetings were held with groups of 
teachers to discuss, first, the initial ideas of the design team, and later some of the more fine tuning needed. 
Many of these teachers were very generous in giving time to try out many of the test items in their embryonic 
stages, and providing feedback. When the first draft of all the tests was finished they were submitted to pilot 
trials in several schools. Feedback from these trials informed further development. Finally a formal trial was 
conducted in a wide range of schools and marking was conducted using the marking key. The data from this 
trial was analysed using the Rasch extended logistic model. As a result of this trial further modifications were 
made to the assessment instruments. 

THE SAMPLE 

The standardized tests were administered system-wide to a random sample of 11,166 students (Education 
Department of Western Australia, 1998) drawn from a wide range of schools across the state of Western 
Australia, including students from metropolitan schools, large and small rural schools, isolated and remote 
schools in desert regions. It was decided that it would be unfair to ask each selected school in the sample to 
complete tests for more than one of the of the arts. So each child only did the tests for one of the disciplines. 
Rather than randomly selecting individual students for testing, as is normally the case, it was decided that 
there would be less disruption if whole class groups were randomly selected. For each of the disciplines at 
years three and seven, forty class groups completed the tests. At year ten, the tests were only administered 
to those students studying the particular discipline. That is, for example, only students studying visual art at 
the time of the test administration completed the visual art tests. Because the pool of arts classes at year ten 
was smaller than that of years three and seven, only twenty classes at this level completed the tests. 

THE VISUAL ART TESTS 

The analysis tests for Visual Art involved students examining their own copies of colour reproductions of 
paintings and responding to questions about the work 




The reproduction of the painting “Waiting for the Broth” 
by George Duerdin was used as the only stimulus for 
the year three Analysis Test, 
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whereas for year seven this reproduction was used 
with another picture, “Family”, by Tania Ferrier. 



In the Year ten test, both of these reproductions were 
used together with a third, “The Merritt Sisters” by Julie 
Dowling (all Western Australian art works). 



In the Process Test students were given a stimulus and asked to produce a coloured drawing. Teachers of 
year three students were given a range of four stimulus ideas from which to choose the most suitable for 
their class. These were: 

1 . Playing my favourite game 

2. Imagine you could have any animal as a pet. What sort of adventure could you 
have? Draw a picture about it. 

3. In a dream you visit a strange land where the trees, the people and the buildings 
are very different. Draw a picture of what you see. 

4. Draw yourself doing the thing you enjoy most at school. 

It was believed important to offer choice for the year three teachers in order to accommodate the special 
needs of junior primary children. The year seven and year ten students were given a short poem with clear 
visual imagery as a stimulus. Although students had their own copy, teachers were requested to read out the 
poem and conduct a short discussion about the poem’s main ideas for those students who may have had 
difficulty with the text, for example, students whose first language is not English. 

Tumbling waters rushing by 
Streaming down a mountain side 
Making its way to the mountain stream 
Of flashing silver. 

Shining liquid shadowed by buildings, 

Channelled into tunnels 
A rampant spirit caged 
And beaten by the unnatural. 

Back to the ocean. 

Dripping out a cement pipe. 

A slow, sad trickle. 

Nicole Stubbs, Aged 15 
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In the instructions to teachers it was emphasised that teachers offer their students a range of colour media 
with which they believed them to be familiar. In early trials of the test teachers were asked to make available 
to students as wide a range of media as possible from which students could make a choice. This proved 
problematic in several ways. In many cases students chose media with which they were not familiar, possibly 
because they were attracted by the novelty of something new. Students tried to use too large a range of 
media in one work and often inappropriate media choices were made, for example, glitter was applied 
unnecessarily. 

As for tests in the other disciplines, visual art students were asked to plan their work on paper provided, 
complete their art work as best as possible in the time provided, then evaluate their own work, responding to 
questions such as the following from the year 10 test. 

1. What parts of your picture do you think are successful? Explain briefly using 
the art elements. 

2. What changes in art elements and principles would you make to your picture if 
you had more time? 

OBSERVATIONS AFTER THE EVENT 

Throughout the process of test development, implementation and marking the design team had complete 
support from the arts teaching body. We relied heavily upon teams of teachers who willingly gave their 
classes to trial our ideas. The arts teaching fraternity generally saw the process as one from which 
something positive may arise to assist the teaching and learning of the arts in Western Australia. As is often 
the case elsewhere, arts teachers generally believe that the arts in education come a poor second to other 
disciplines and teachers expressed the hope that that this situation may improve because as a result of 
these tests the Education Department of Western Australia would now have documented evidence of 
standards of learning in the arts. This evidence would point to strengths and shortcomings. Teachers 
expressed the hope that this evidence would lead to an improvement in conditions for teaching and learning 
and an improvement in status for the arts in education. This has certainly been the case after assessments in 
other learning areas and it only seems reasonable to expect that the arts will receive similar attention. As the 
report of the assessment (Education Department of Western Australia, 1998) was only released twelve 
months ago it is probably too soon to observe any positive outcomes. 

SOME INTERESTING OUTCOMES 

The assessment report (Education Department of Western Australia, 1998) produced by the Monitoring 
Standards includes only statistics without any analysis of the data. However, the data included is 
comprehensive allowing information to be drawn from it. For example, the following interesting facts emerge. 

• In every test for every arts discipline for every year level girls significantly out-performed boys. 

• Apart from the year ten Drama assessment, Aboriginal and Torres Strait Islander students achieved 
significantly less than other groups in every test. 

• The data for the Dance and Music assessments reveals a significant improvement in student performance 
between years seven and ten, whereas in Visual Art, there is relatively little improvement in student 
performance between those years. Although a variety of explanations may be offered, such as some year 
ten students studying Visual Art for the first time in the semester the assessment was administered, or there 
being no expectation or requirement of continuous study in the Visual Art for the accumulation of skills and 
knowledge (as there is in Dance and Music). Such a result could provide a concern for Visual Art education 
in Western Australia. 

• When it is known that in Western Australian schools, at the time of this assessment, students received little 
or no tuition in Dance of Media, how was it possible for these students to show marked improvement in skills 
from one level to the next? Perhaps answers may be found in the marking key, or the ease with which 
markers gave marks for student effort, or the fact that the Outcome Statements - the benchmarks against 
which the student achievement was matched - are too generic, not discerning enough and do not adequately 
describe expected achievement at each of the eight outcome levels. I am concerned that much of what has 
been assessed and assigned an outcome level in this assessment is the developing natural ability of the 
student, rather than ability and knowledge developed by good teaching and learning environments. 

• For the purpose of the report writing, data from the process tests and data from the analysis tests were 
combined. Consequently the report gives no separate information about how students performed in making 
art compared to how they understand and appreciate art. However, a separate study of the data has 
revealed that students achieved significantly better, at all levels, in the Process Tests than they did in the 
Analysis Tests. Likewise, in those aspects of the Process Tests which required written language, that is, the 
planning and critique stages, students scored poorly. This aspect of the results points to a problem in 
teaching and learning in the arts. It seems that students in Western Australia are not able to write about art, 
that is, their own art or the art of others, with the same ability with which they are able to make art. This state 
of affairs may be turned around with the new Outcome Frameworks currently being put in place in our 
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schools. For students to be able to proceed satisfactorily through the different outcome levels they will need 
to be exposed to a balanced arts education program. Teachers will have to address both the making of art 
together with the appreciation of art. For many teachers, especially in primary schools, this could mean the 
need for major professional development programs to be put in place by educational authorities. 

There is one other comment I would like to make about an observation made during the process of test 
development. This was not something that showed up in the data, rather, it was something that emerged 
during the trials of test strategies. It became obvious that specialist teachers of Visual Art and Drama, not 
Media, Music or Dance, were very reluctant to ‘let go’ of their students and allow them to complete the tests 
from start to finish without teacher input. Teachers complained that the tests should follow normal classroom 
practice and have the teacher lead the students from initial stimulus through to the final product. These 
teachers saw it as a natural process that they should remain a part of their student art making throughout the 
process. When teachers did ‘let go’ and follow the test instructions, they were clearly disappointed with their 
students’ products, explaining that normally their students produced efforts of a much higher standard. One 
might question, therefore, whose art it really is that is attributed to the students in “normal classroom 
practice”. Why is the assessment of Visual Art and Drama so different from that of Mathematics, English, 
Science and other subjects? In assessment of these subjects it is a normal expectation that students 
complete tests without the assistance of the teacher. What is different about Visual Art and Drama that this is 
not the case in their assessment? 

CONCLUSION 

The title of this paper includes the words “a developing model”. It is expected that the Education Department 
will undergo a second round of monitoring standards in the arts in the year 2001. My hope is that the model 
we have developed for the 1996 assessment will be developed further to extend those administered in 1996 
so we can see that there is progress in our students’ achievement. The 1996 tests were always seen as a 
modest and simple starting point. We now have a firm base from which to expand and move forward with 
some confidence. 
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